Skip to content

logs_to_metrics: Support optional value_field for counters #10055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

logs_to_metrics: Support optional value_field for counters #10055

wants to merge 1 commit into from

Conversation

databus23
Copy link

@databus23 databus23 commented Mar 6, 2025 •
edited
Loading

Support incrementing counters by a specified value

If a value_field is specified for a counter metric, the counter is incremented by the given value instead of just incrementing by 1.

This allows counting more things then just the number of records, such as the total number of bytes sent or received from an access log.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

  • Documentation required for this feature

Doc PR: fluent/fluent-bit-docs#1584

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@databus23 databus23 force-pushed the logs_to_metric_counter_value branch 2 times, most recently from 68cd15e to e725188 Compare March 6, 2025 23:09
@databus23 databus23 changed the title logs_to_metric: Support optional value_field for counters logs_to_metrics: Support optional value_field for counters Mar 6, 2025
@databus23 databus23 force-pushed the logs_to_metric_counter_value branch from e725188 to 2c1822f Compare March 6, 2025 23:12
@databus23
logs_to_metrics: Support optional value_field for counters
db182ed 
Support incrementing counters by a specified value.

If a `value_field` is specified for a counter, the counter is incremented by the given value instead of just counting the number of records.

This allows tracking different metrics, such as the total number of bytes sent or received in an access log.

Signed-off-by: Fabian Ruff <[email protected]>
@databus23 databus23 force-pushed the logs_to_metric_counter_value branch from 2c1822f to db182ed Compare March 9, 2025 00:10
@databus23
Copy link
Author

databus23 commented Mar 10, 2025 •
edited
Loading

Valgrind output for unit tests:

fluent-bit ➜ /workspaces/fluent-bit/build (logs_to_metric_counter_value) $ valgrind bin/flb-rt-filter_log_
to_metrics 
==35868== Memcheck, a memory error detector
==35868== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==35868== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==35868== Command: bin/flb-rt-filter_log_to_metrics
==35868== 
Test counter_k8s...                             [ OK ]
Test counter...                                 [ OK ]
Test counter_value_field...                     [ OK ]
Test counter_k8s_two_tuples...                  [ OK ]
Test gauge...                                   [ OK ]
Test histogram...                               [ OK ]
Test counter_regex...                           [ OK ]
Test regex_empty_label_keys...                  [ OK ]
Test label...                                   [ OK ]
SUCCESS: All unit tests have passed.
==35868== 
==35868== HEAP SUMMARY:
==35868==     in use at exit: 0 bytes in 0 blocks
==35868==   total heap usage: 22,585 allocs, 22,585 frees, 16,092,238 bytes allocated
==35868== 
==35868== All heap blocks were freed -- no leaks are possible
==35868== 
==35868== For lists of detected and suppressed errors, rerun with: -s
==35868== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

databus23 added a commit to databus23/fluent-bit-docs that referenced this pull request Mar 10, 2025
@databus23
filter_logs_to_metrics: document field_value usage for counters
dfaf16d 
This documents a feature added with this PR: fluent/fluent-bit#10055

Signed-off-by: Fabian Ruff <[email protected]>
@databus23 databus23 mentioned this pull request Mar 10, 2025
Open
@databus23
Copy link
Author

@edsiper Kindly ping. At least the docs-required label can be removed

@databus23
Copy link
Author

@leonardo-albertovich kindly ping. Any chance to get a response

@databus23
Copy link
Author

@koleini @fujimotos Kindly ping. Any change that is is being looked at? It has tests, docs, all the things. I would love to see some progress on my PR. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@databus23